{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cd7996c5-91eb-41de-ac6a-0b22462b76f1",
   "metadata": {},
   "source": [
    "# 第五章 自动合并检索"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1018f912",
   "metadata": {},
   "source": [
    "在本课程中，我们将探索一种高级的检索聚合（RAG）技术——自动合并检索。这种方法解决了使用传统RAG时遇到的一个关键问题：当检索信息块以填充语言模型（LLM）上下文时，如果这些块过于碎片化，将会影响上下文的连贯性，特别是当这些块的大小较小时。为了克服这一挑战，我们将采用一种自动合并启发式方法，该方法通过将较小的信息块合并到更大的父块中，从而保证了上下文的连贯性。接下来，我们将详细介绍如何实现这一技术，并通过实际案例来演示其应用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "144900d8",
   "metadata": {},
   "source": [
    "首先，我们需要准备环境和导入必要的库。请注意，为了避免不必要的警告信息干扰我们的操作，我们将使用warnings库来忽略它们。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e8f0928d-4505-437c-9ee8-bada425a5e22",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfeb0fa8",
   "metadata": {},
   "source": [
    "接下来，我们将导入一些工具函数，设置OpenAI的API密钥，并准备我们的文档。这些文档将作为我们的检索源。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "20373372-03c5-41e9-8bc9-dfa6ce35d666",
   "metadata": {
    "height": 98
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ In Answer Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In Answer Relevance, input response will be set to __record__.main_output or `Select.RecordOutput` .\n",
      "✅ In Context Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In Context Relevance, input response will be set to __record__.app.query.rets.source_nodes[:].node.text .\n",
      "✅ In Groundedness, input source will be set to __record__.app.query.rets.source_nodes[:].node.text .\n",
      "✅ In Groundedness, input statement will be set to __record__.main_output or `Select.RecordOutput` .\n"
     ]
    }
   ],
   "source": [
    "import utils\n",
    "\n",
    "import os\n",
    "import openai\n",
    "# 通过utils获取OpenAI API密钥并设置\n",
    "openai.api_key = utils.get_openai_api_key()\n",
    "\n",
    "# 或者这里填入你的OpenAI API密钥\n",
    "# openai.api_key = \"\" \n",
    "\n",
    "# 或者自定义API密钥和API基础地址，可适用第三方API服务\n",
    "# openai.api_key = \"sk- \"  \n",
    "# openai.api_base = \" \""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8388024",
   "metadata": {},
   "source": [
    "我们将使用 `llama_index` 库来读取和处理文档。在这个例子中，我们将加载一个PDF文件，但鼓励大家尝试使用自己的文档。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f58fc195-9619-4088-831e-66b3cc6e0425",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index import SimpleDirectoryReader\n",
    "\n",
    "documents = SimpleDirectoryReader(\n",
    "    input_files=[\"./data/人工智能.pdf\"]\n",
    ").load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c342b21f",
   "metadata": {},
   "outputs": [],
   "source": [
    "documents_en = SimpleDirectoryReader(\n",
    "    input_files=[\"./data/eBook-How-to-Build-a-Career-in-AI.pdf\"]\n",
    ").load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9c6f37a",
   "metadata": {},
   "source": [
    "为了验证我们的文档已经被正确加载，让我们打印出文档的类型、数量以及第一个文档的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "89ad555b-48d4-441b-b5f3-79a685b4c3f2",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> \n",
      "\n",
      "7 \n",
      "\n",
      "<class 'llama_index.schema.Document'>\n",
      "Doc ID: 35bf3256-2c52-4995-ae3b-311956872365\n",
      "Text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称 中国⼤陆⼈⼯智能 台湾⼈⼯智慧\n",
      "港澳⼈⼯智能 新⻢⼈⼯智能、⼈⼯智慧 ⽇韩⼈⼯知能 越南智慧⼈造 [展开] [展开] [展开] [展开] [展开] [展开]⼈⼯智能系列内容\n",
      "主要⽬标 实现⽅式 ⼈⼯智能哲学 历史 技术 术语⼈⼯智能（英语：artiﬁcial intelligence ，缩写为\n",
      "AI）亦称机器智能，指由⼈制造出来的机器所表现出来的智能。通常⼈⼯\n",
      "智能是指⽤普通计算机程序来呈现⼈类智能的技术。该词也指出研究这样的智能系统是否能够实现，以及如何实现。同 时，通过 医学 、神经科学\n",
      "、机器⼈学 及...\n"
     ]
    }
   ],
   "source": [
    "print(type(documents), \"\\n\")\n",
    "print(len(documents), \"\\n\")\n",
    "print(type(documents[0]))\n",
    "print(documents[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4d2a19d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> \n",
      "\n",
      "41 \n",
      "\n",
      "<class 'llama_index.schema.Document'>\n",
      "Doc ID: ab54b5ce-57c8-4187-91c4-de8c3d44cf4e\n",
      "Text: PAGE 1Founder, DeepLearning.AICollected Insights from Andrew Ng\n",
      "How to  Build Your Career in AIA Simple Guide\n"
     ]
    }
   ],
   "source": [
    "print(type(documents_en), \"\\n\")\n",
    "print(len(documents_en), \"\\n\")\n",
    "print(type(documents_en[0]))\n",
    "print(documents_en[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c143850-72c6-4a09-9fb0-a216108314c0",
   "metadata": {},
   "source": [
    "## 一、自动合并检索的设置"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d90c5394",
   "metadata": {},
   "source": [
    "我们将所有文档合并成一个大的文档对象，以便进行自动合并检索。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8926348c-a4c5-471b-99ed-e6bf170f1fe1",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index import Document\n",
    "\n",
    "# 将所有文档文本合并成一个大文档\n",
    "document = Document(text=\"\\n\\n\".join([doc.text for doc in documents]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4af9556f",
   "metadata": {},
   "outputs": [],
   "source": [
    "document_en = Document(text=\"\\n\\n\".join([doc.text for doc in documents_en]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bb7650eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将中文标点符号替换成英文标点符号，方便后续处理\n",
    "# 如果是英文文档，可以跳过这一步\n",
    "# 不处理的话，会导致无法正确切分中文句子，会影响后续sentence_window的大小，导致输入长度大于gpt-3.5-turbo的最大限制\n",
    "document.text=document.text.replace('。','. ')\n",
    "document.text=document.text.replace('！','! ')\n",
    "document.text=document.text.replace('？','? ')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa6cb081",
   "metadata": {},
   "source": [
    "现在，我们将创建一个层次结构节点解析器。这个解析器将帮助我们根据文档内容创建一个层次结构，以便于后续的自动合并操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "202ccfb5-cf94-48a4-b23b-8c5a5ab73ebc",
   "metadata": {
    "height": 115,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.node_parser import HierarchicalNodeParser\n",
    "\n",
    "# 使用默认设置创建层次结构节点解析器\n",
    "node_parser = HierarchicalNodeParser.from_defaults(\n",
    "    chunk_sizes=[2048, 512, 128]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16be817c",
   "metadata": {},
   "source": [
    "通过节点解析器，我们得到了文档中的所有节点。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "672fd574-6b3c-43ad-9a9a-2db423cadd48",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 从文档中获取节点\n",
    "nodes = node_parser.get_nodes_from_documents([document])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "052c5009",
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes_en = node_parser.get_nodes_from_documents([document_en])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4027648",
   "metadata": {},
   "source": [
    "为了展示如何具体操作，我们将取出一些叶节点并查看它们的内容。叶节点是层次结构中最小的块，它们将直接参与到后续的合并过程中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "83cfc301-1887-4b1c-a8aa-1fb55e30be78",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⼈⼯智能⼯具\n",
      "与司法部门的⼈类法官相辅相成，提供客观、⼀致的风险评估[13].\n"
     ]
    }
   ],
   "source": [
    "from llama_index.node_parser import get_leaf_nodes\n",
    "\n",
    "# 获取叶节点并打印其中一个的文本作为示例\n",
    "leaf_nodes = get_leaf_nodes(nodes)\n",
    "print(leaf_nodes[30].text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2d8f6d85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "But this became less important as numerical linear algebra libraries matured.\n",
      "Deep learning is still an emerging technology, so when you train a neural network and the \n",
      "optimization algorithm struggles to converge, understanding the math behind gradient \n",
      "descent, momentum, and the Adam  optimization algorithm will help you make better decisions. \n",
      "Similarly, if your neural network does something funny — say, it makes bad predictions on \n",
      "images of a certain resolution, but not others — understanding the math behind neural network \n",
      "architectures puts you in a better position to figure out what to do.\n",
      "Of course, I also encourage learning driven by curiosity.\n"
     ]
    }
   ],
   "source": [
    "leaf_nodes_en = get_leaf_nodes(nodes_en)\n",
    "print(leaf_nodes_en[30].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d386fd2",
   "metadata": {},
   "source": [
    "我们也可以查看这些叶节点的父节点。父节点包含了它的所有子节点的内容，这对于我们理解自动合并的过程非常重要。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0703abeb-3bcc-4dd6-8010-3bcf95d8b483",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⼈⼯智能为地⽅政府服务带来技术突破. ⼈⼯智能代理协助城市规划者基于⽬标导向的蒙特卡罗树搜索进⾏场景规划. ⽬标推理⼈⼯智能代理提供最佳的⼟\n",
      "地利⽤解决⽅案，帮助我们制定民主的城市⼟地利⽤规划. ⼈⼯智能利⽤在线数据来监控和修改环境威胁政策. 在2019 年⽔危机期间，潜在狄利克雷分配⽅\n",
      "法确定了Twitter (X) 中讨论最多的主题，这是⼀种朴素的推⽂分类⽅法，对⼲旱的影响和原因、政府响应和潜在解决⽅案等主题进⾏了分类. ⼈⼯智能⼯具\n",
      "与司法部门的⼈类法官相辅相成，提供客观、⼀致的风险评估[13].\n"
     ]
    }
   ],
   "source": [
    "# 根据ID获取节点，并打印一个叶节点的父节点文本作为示例\n",
    "nodes_by_id = {node.node_id: node for node in nodes}\n",
    "\n",
    "parent_node = nodes_by_id[leaf_nodes[30].parent_node.node_id]\n",
    "print(parent_node.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2927ce52",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PAGE 12Should You \n",
      "Learn Math to \n",
      "Get a Job in AI? CHAPTER 3\n",
      "LEARNING\n",
      "\n",
      "PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? It’s always nice to know more math! But there’s so much to \n",
      "learn that, realistically, it’s necessary to prioritize. Here’s how you might go about strengthening \n",
      "your math background.\n",
      "To figure out what’s important to know, I find it useful to ask what you need to know to make \n",
      "the decisions required for the work you want to do. At DeepLearning.AI, we frequently ask, \n",
      "“What does someone need to know to accomplish their goals?” The goal might be building a \n",
      "machine learning model, architecting a system, or passing a job interview.\n",
      "Understanding the math behind algorithms you use is often helpful, since it enables you to \n",
      "debug them. But the depth of knowledge that’s useful changes over time. As machine learning \n",
      "techniques mature and become more reliable and turnkey, they require less debugging, and a \n",
      "shallower understanding of the math involved may be sufficient to make them work.\n",
      "For instance, in an earlier era of machine learning, linear algebra libraries for solving linear \n",
      "systems of equations (for linear regression) were immature. I had to understand how these \n",
      "libraries worked so I could choose among different libraries and avoid numerical roundoff \n",
      "pitfalls. But this became less important as numerical linear algebra libraries matured.\n",
      "Deep learning is still an emerging technology, so when you train a neural network and the \n",
      "optimization algorithm struggles to converge, understanding the math behind gradient \n",
      "descent, momentum, and the Adam  optimization algorithm will help you make better decisions. \n",
      "Similarly, if your neural network does something funny — say, it makes bad predictions on \n",
      "images of a certain resolution, but not others — understanding the math behind neural network \n",
      "architectures puts you in a better position to figure out what to do.\n",
      "Of course, I also encourage learning driven by curiosity. If something interests you, go ahead \n",
      "and learn it regardless of how useful it might turn out to be!  Maybe this will lead to a creative \n",
      "spark or technical breakthrough.How much math do you need to know to be a machine learning engineer?\n"
     ]
    }
   ],
   "source": [
    "nodes_by_id_en = {node.node_id: node for node in nodes_en}\n",
    "\n",
    "parent_node_en = nodes_by_id_en[leaf_nodes_en[30].parent_node.node_id]\n",
    "print(parent_node_en.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6bafef6-02f6-4029-a5eb-dfbc4369da56",
   "metadata": {},
   "source": [
    "### 1.1、构建索引"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db3656fc",
   "metadata": {},
   "source": [
    "我们将构建索引并定义检索器，以便运行查询引擎并评估自动合并检索的效果。\n",
    "\n",
    "继续构建我们的自动合并检索系统，我们需要使用 OpenAI 的语言模型和一些自定义组件来构建索引，这些组件将支持我们的自动合并逻辑。\n",
    "\n",
    "首先，我们将初始化一个 OpenAI 的语言模型，这里我们选择使用 GPT-3.5 Turbo ，这是一个强大的模型，适用于各种NLP任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6b0350cd-392e-45e9-8664-ff22a39a8bcd",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.llms import OpenAI\n",
    "\n",
    "# 使用OpenAI的GPT-3.5 Turbo模型\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16e47cf1",
   "metadata": {},
   "source": [
    "接下来，我们将创建一个服务上下文对象。这个对象将包含我们所需的所有服务，例如嵌入模型和节点解析器，以便我们的系统能够正确地执行自动合并检索。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "03213595-6581-47f5-90a7-cfa381d42872",
   "metadata": {
    "height": 132,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index import ServiceContext\n",
    "\n",
    "# 创建服务上下文，包括LLM模型和节点解析器\n",
    "auto_merging_context = ServiceContext.from_defaults(\n",
    "    llm=llm,\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",\n",
    "    node_parser=node_parser,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "401210d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 英文 embeding 模型\n",
    "auto_merging_context_en = ServiceContext.from_defaults(\n",
    "    llm=llm,\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",\n",
    "    node_parser=node_parser,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "126c65ba",
   "metadata": {},
   "source": [
    "现在，我们需要将我们的文档节点存储在一个索引中。这个索引将支持快速检索操作，并允许我们根据需要动态合并文档节点。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a0f84886-e1aa-40f1-a812-4b6f8d0204af",
   "metadata": {
    "height": 198,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index import VectorStoreIndex, StorageContext\n",
    "\n",
    "# 创建存储上下文并添加文档节点\n",
    "storage_context = StorageContext.from_defaults()\n",
    "storage_context.docstore.add_documents(nodes)\n",
    "\n",
    "# 创建自动合并索引\n",
    "automerging_index = VectorStoreIndex(\n",
    "    leaf_nodes, storage_context=storage_context, service_context=auto_merging_context\n",
    ")\n",
    "\n",
    "# 持久化索引以便未来使用\n",
    "automerging_index.storage_context.persist(persist_dir=\"./merging_index\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "314e5cd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 创建存储上下文并添加文档节点\n",
    "storage_context_en = StorageContext.from_defaults()\n",
    "storage_context_en.docstore.add_documents(nodes_en)\n",
    "\n",
    "# 创建自动合并索引\n",
    "automerging_index_en = VectorStoreIndex(\n",
    "    leaf_nodes_en, storage_context=storage_context_en, service_context=auto_merging_context_en\n",
    ")\n",
    "\n",
    "# 持久化索引以便未来使用\n",
    "automerging_index_en.storage_context.persist(persist_dir=\"./merging_index_en\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ed2848c",
   "metadata": {},
   "source": [
    "为了确保我们的系统能够在需要时重新加载索引，我们还提供了一个可选的代码块，用于检查索引文件是否存在，并根据需要加载或重新构建它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4a8ddbff-6fdd-448d-9ce8-53ed782a6d66",
   "metadata": {
    "height": 453,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 可选代码块：检查索引文件是否存在，如果不存在则重建\n",
    "import os\n",
    "from llama_index import VectorStoreIndex, StorageContext, load_index_from_storage\n",
    "from llama_index import load_index_from_storage\n",
    "\n",
    "if not os.path.exists(\"./merging_index\"):\n",
    "    storage_context = StorageContext.from_defaults()\n",
    "    storage_context.docstore.add_documents(nodes)\n",
    "\n",
    "    automerging_index = VectorStoreIndex(\n",
    "            leaf_nodes,\n",
    "            storage_context=storage_context,\n",
    "            service_context=auto_merging_context\n",
    "        )\n",
    "\n",
    "    automerging_index.storage_context.persist(persist_dir=\"./merging_index\")\n",
    "else:\n",
    "    automerging_index = load_index_from_storage(\n",
    "        StorageContext.from_defaults(persist_dir=\"./merging_index\"),\n",
    "        service_context=auto_merging_context\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0a97cdd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists(\"./merging_index_en\"):\n",
    "    storage_context_en = StorageContext.from_defaults()\n",
    "    storage_context_en.docstore.add_documents(nodes_en)\n",
    "\n",
    "    automerging_index_en = VectorStoreIndex(\n",
    "            leaf_nodes_en,\n",
    "            storage_context=storage_context_en,\n",
    "            service_context=auto_merging_context_en\n",
    "        )\n",
    "\n",
    "    automerging_index_en.storage_context_en.persist(persist_dir=\"./merging_index_en\")\n",
    "else:\n",
    "    automerging_index_en = load_index_from_storage(\n",
    "        StorageContext.from_defaults(persist_dir=\"./merging_index_en\"),\n",
    "        service_context=auto_merging_context_en\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2710c689-0596-45c5-a63f-1e8bb9481846",
   "metadata": {},
   "source": [
    "### 1.2、定义检索器并运行查询引擎"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "464db5ee",
   "metadata": {},
   "source": [
    "接下来，我们将定义检索器并运行查询引擎。这个过程中，我们将设置一个自动合并检索器，这个检索器将控制文档节点的合并逻辑，以确保检索到的上下文是连贯的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d443d25a-2a5e-425c-9cca-4bf8503c6db3",
   "metadata": {
    "height": 351,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.indices.postprocessor import SentenceTransformerRerank\n",
    "from llama_index.retrievers import AutoMergingRetriever\n",
    "from llama_index.query_engine import RetrieverQueryEngine\n",
    "\n",
    "# 将自动合并索引转换为检索器\n",
    "automerging_retriever = automerging_index.as_retriever(\n",
    "    similarity_top_k=12\n",
    ")\n",
    "\n",
    "# 定义自动合并检索器\n",
    "retriever = AutoMergingRetriever(\n",
    "    automerging_retriever, \n",
    "    automerging_index.storage_context, \n",
    "    verbose=True\n",
    ")\n",
    "\n",
    "# 定义重新排名模块\n",
    "rerank = SentenceTransformerRerank(top_n=6, model=\"BAAI/bge-reranker-base\")\n",
    "\n",
    "# 创建自动合并检索查询引擎\n",
    "auto_merging_engine = RetrieverQueryEngine.from_args(\n",
    "    automerging_retriever, node_postprocessors=[rerank]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b77563d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将自动合并索引转换为检索器\n",
    "automerging_retriever_en = automerging_index_en.as_retriever(\n",
    "    similarity_top_k=12\n",
    ")\n",
    "\n",
    "# 定义自动合并检索器\n",
    "retriever_en = AutoMergingRetriever(\n",
    "    automerging_retriever_en, \n",
    "    automerging_index_en.storage_context, \n",
    "    verbose=True\n",
    ")\n",
    "\n",
    "# 定义重新排名模块\n",
    "rerank_en = SentenceTransformerRerank(top_n=6, model=\"BAAI/bge-reranker-base\")\n",
    "\n",
    "# 创建自动合并检索查询引擎\n",
    "auto_merging_engine_en = RetrieverQueryEngine.from_args(\n",
    "    automerging_retriever_en, node_postprocessors=[rerank_en]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "003c2954",
   "metadata": {},
   "source": [
    "为了测试我们的自动合并检索系统，我们将使用一个示例查询：“AI对人类的威胁有哪些？”并展示检索到的响应。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c458d0ac-24a0-430b-acc4-c05252134527",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 使用查询引擎执行查询\n",
    "auto_merging_response = auto_merging_engine.query(\n",
    "    \"AI对人类的威胁有哪些？\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "616d4c77",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用查询引擎执行查询\n",
    "auto_merging_response_en = auto_merging_engine_en.query(\n",
    "    \"What is the importance of networking in AI?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cc84276",
   "metadata": {},
   "source": [
    "我们可以使用提供的工具函数来美观地展示检索到的响应。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "569e70b4-daae-4181-a953-7ea37f4620c7",
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**`Final Response:`** AI对人类的威胁包括：AI会遵循科技发展的加速度理论、AI可能会有自我改造创新的能力、AI进步的速度远远超过人类、人类会有被灭绝的危机。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from llama_index.response.notebook_utils import display_response\n",
    "\n",
    "# 显示查询响应\n",
    "display_response(auto_merging_response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "9cf2d408",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**`Final Response:`** Networking in AI is crucial as it allows individuals to build a strong professional community that can provide valuable information, support, and opportunities. By connecting with others in the field, individuals can receive guidance, advice, and potential referrals to employers. Additionally, networking helps individuals stay updated on the latest trends and developments in AI, fostering continuous learning and growth within the community."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_response(auto_merging_response_en)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a841036a-0bb6-4037-920e-ae555748e111",
   "metadata": {},
   "source": [
    "## 二、整合所有步骤"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "740c4160",
   "metadata": {},
   "source": [
    "我们将整合上述所有步骤，并通过 TruLens 对我们的自动合并检索系统进行评估，以验证其性能。\n",
    "\n",
    "为了将自动合并检索的各个部分整合在一起并进行评估，我们将定义两个高级函数：`build_automerging_index` 和 `get_automerging_query_engine` 。这些函数将帮助我们构建索引并初始化查询引擎，使我们能够以更简洁的方式测试和评估不同的配置。\n",
    "\n",
    "首先，我们定义 `build_automerging_index` 函数，该函数负责构建自动合并索引。这个函数将接受文档、语言模型、嵌入模型和保存目录作为参数，并构建一个包含所有文档节点的索引。\n",
    "\n",
    "接下来，我们定义 `get_automerging_query_engine` 函数，该函数负责从自动合并索引中创建查询引擎。它设置了检索器，使之能够根据相似性的 top K 值进行检索，并应用重新排序逻辑来优化检索结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "8949da64-bc7f-4929-b394-51d11732e62f",
   "metadata": {
    "height": 1099
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from llama_index import (\n",
    "    ServiceContext,\n",
    "    StorageContext,\n",
    "    VectorStoreIndex,\n",
    "    load_index_from_storage,\n",
    ")\n",
    "from llama_index.node_parser import HierarchicalNodeParser\n",
    "from llama_index.node_parser import get_leaf_nodes\n",
    "from llama_index import StorageContext, load_index_from_storage\n",
    "from llama_index.retrievers import AutoMergingRetriever\n",
    "from llama_index.indices.postprocessor import SentenceTransformerRerank\n",
    "from llama_index.query_engine import RetrieverQueryEngine\n",
    "\n",
    "\n",
    "# 定义构建自动合并索引的函数\n",
    "def build_automerging_index(\n",
    "    documents,\n",
    "    llm,\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",\n",
    "    save_dir=\"merging_index\",\n",
    "    chunk_sizes=None,\n",
    "):\n",
    "    chunk_sizes = chunk_sizes or [2048, 512, 128]\n",
    "    node_parser = HierarchicalNodeParser.from_defaults(chunk_sizes=chunk_sizes)\n",
    "    nodes = node_parser.get_nodes_from_documents(documents)\n",
    "    leaf_nodes = get_leaf_nodes(nodes)\n",
    "    merging_context = ServiceContext.from_defaults(\n",
    "        llm=llm,\n",
    "        embed_model=embed_model,\n",
    "    )\n",
    "    storage_context = StorageContext.from_defaults()\n",
    "    storage_context.docstore.add_documents(nodes)\n",
    "\n",
    "    if not os.path.exists(save_dir):\n",
    "        automerging_index = VectorStoreIndex(\n",
    "            leaf_nodes, storage_context=storage_context, service_context=merging_context\n",
    "        )\n",
    "        automerging_index.storage_context.persist(persist_dir=save_dir)\n",
    "    else:\n",
    "        automerging_index = load_index_from_storage(\n",
    "            StorageContext.from_defaults(persist_dir=save_dir),\n",
    "            service_context=merging_context,\n",
    "        )\n",
    "    return automerging_index\n",
    "\n",
    "\n",
    "# 定义获取自动合并查询引擎的函数\n",
    "def get_automerging_query_engine(\n",
    "    automerging_index,\n",
    "    similarity_top_k=12,\n",
    "    rerank_top_n=6,\n",
    "):\n",
    "    base_retriever = automerging_index.as_retriever(similarity_top_k=similarity_top_k)\n",
    "    retriever = AutoMergingRetriever(\n",
    "        base_retriever, automerging_index.storage_context, verbose=True\n",
    "    )\n",
    "    rerank = SentenceTransformerRerank(\n",
    "        top_n=rerank_top_n, model=\"BAAI/bge-reranker-base\"\n",
    "    )\n",
    "    auto_merging_engine = RetrieverQueryEngine.from_args(\n",
    "        retriever, node_postprocessors=[rerank]\n",
    "    )\n",
    "    return auto_merging_engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16d44525",
   "metadata": {},
   "source": [
    "使用这两个函数，我们可以方便地构建索引和查询引擎，并通过不同的参数配置来测试自动合并检索的效果。下面是如何使用这些函数的示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b7f52577-2e8e-4fee-ade9-2f5ce96a7a8c",
   "metadata": {
    "height": 149
   },
   "outputs": [],
   "source": [
    "from llama_index.llms import OpenAI\n",
    "\n",
    "# 构建自动合并索引\n",
    "index = build_automerging_index(\n",
    "    [document],\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    save_dir=\"./merging_index\",\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "455a917c",
   "metadata": {},
   "outputs": [],
   "source": [
    "index_en = build_automerging_index(\n",
    "    [document_en],\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    save_dir=\"./merging_index_en\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "c753885f-bfaf-4ac0-8410-b62f619827c4",
   "metadata": {
    "height": 45
   },
   "outputs": [],
   "source": [
    "# 获取自动合并查询引擎\n",
    "query_engine = get_automerging_query_engine(index, similarity_top_k=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "7dcfb680",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine_en = get_automerging_query_engine(index_en, similarity_top_k=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9130e35d-bc09-4e4d-bbd9-61c9d6847110",
   "metadata": {},
   "source": [
    "## 三、TruLens评估"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09e44f84",
   "metadata": {},
   "source": [
    "在构建了自动合并索引和查询引擎后，我们将使用 TruLens 来评估自动合并检索器的性能。 TruLens 是一个用于评估 AI 模型的工具，可以提供详细的性能指标。我们将首先重置 TruLens 数据库，然后进行评估。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "cf034319-2f0a-43ac-896b-2aef1762b3bd",
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🦑 Tru initialized with db url sqlite:///default.sqlite .\n",
      "🛑 Secret keys may be written to the database. See the `database_redact_keys` option of `Tru` to prevent this.\n"
     ]
    }
   ],
   "source": [
    "from trulens_eval import Tru\n",
    "\n",
    "# 重置TruLens数据库\n",
    "Tru().reset_database()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8515527d",
   "metadata": {},
   "source": [
    "我们将分别评估两层和三层自动合并结构的性能，并使用 TruLens 来记录和展示评估结果。这将帮助我们理解不同层次结构配置对性能的影响，并为我们提供优化自动合并检索策略的洞见。\n",
    "\n",
    "在进行 TruLens 评估之前，我们需要为两种不同的层次结构设置自动合并索引和查询引擎。这两种结构分别是两层结构和三层结构。通过比较这两种结构的性能，我们可以更好地理解层次结构深度对自动合并检索效果的影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76e8c8b7-8d47-45cb-94f3-70c2d75c9500",
   "metadata": {},
   "source": [
    "### 3.1、两层结构的索引构建"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fe6fdfc",
   "metadata": {},
   "source": [
    "首先，我们将构建一个只有两层的自动合并索引。这种结构简化了创建索引的过程，并减少了检索步骤中所需的工作量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "6fd32404-2760-420d-b406-61d50c5ae7de",
   "metadata": {
    "height": 132
   },
   "outputs": [],
   "source": [
    "# 构建具有两层结构的自动合并索引\n",
    "auto_merging_index_0 = build_automerging_index(\n",
    "    documents,\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",\n",
    "    save_dir=\"merging_index_0\",\n",
    "    chunk_sizes=[2048,512],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "522ea409",
   "metadata": {},
   "outputs": [],
   "source": [
    "auto_merging_index_0_en = build_automerging_index(\n",
    "    documents_en,\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",\n",
    "    save_dir=\"merging_index_0_en\",\n",
    "    chunk_sizes=[2048,512],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e9d0180",
   "metadata": {},
   "source": [
    "接着，我们为这个两层结构的自动合并索引设置查询引擎，并保持与之前相同的 top K 和重新排名步骤。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ad319255-b86a-4d05-8dd2-a8fa02d15ed0",
   "metadata": {
    "height": 98
   },
   "outputs": [],
   "source": [
    "# 获取两层结构的自动合并查询引擎\n",
    "auto_merging_engine_0 = get_automerging_query_engine(\n",
    "    auto_merging_index_0,\n",
    "    similarity_top_k=12,\n",
    "    rerank_top_n=6,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "c1ceb480",
   "metadata": {},
   "outputs": [],
   "source": [
    "auto_merging_engine_0_en = get_automerging_query_engine(\n",
    "    auto_merging_index_0_en,\n",
    "    similarity_top_k=12,\n",
    "    rerank_top_n=6,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e48605b",
   "metadata": {},
   "source": [
    "为了进行 TruLens 评估，我们需要定义一个 TruLens 录音器，它将记录查询引擎的响应和评估结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c669cee9-7be9-4a4d-8c17-1672bb9c3abf",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "from utils import get_prebuilt_trulens_recorder\n",
    "\n",
    "# 使用TruLens记录器准备评估\n",
    "tru_recorder = get_prebuilt_trulens_recorder(\n",
    "    auto_merging_engine_0,\n",
    "    app_id ='app_0'  # 设置应用ID\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "ed0a46a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用TruLens记录器准备评估\n",
    "tru_recorder_en = get_prebuilt_trulens_recorder(\n",
    "    auto_merging_engine_0_en,\n",
    "    app_id ='app_0_en'  # 设置应用ID\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faa098ae",
   "metadata": {},
   "source": [
    "然后，我们从预先准备好的问题文件中加载评估问题，并定义一个运行评估的函数。这个函数将遍历所有评估问题，并记录每个问题的查询结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "ad216bb7-100e-455f-8c33-aa68b3e42026",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "# 加载评估问题\n",
    "eval_questions = []\n",
    "with open('./data/generated_questions.txt', 'r', encoding=\"utf8\") as file:\n",
    "    for line in file:\n",
    "        item = line.strip()  # 移除换行符\n",
    "        eval_questions.append(item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "21874b4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 加载评估问题\n",
    "eval_questions_en = []\n",
    "with open('./data/generated_questions_en.txt', 'r') as file:\n",
    "    for line in file:\n",
    "        item = line.strip()  # 移除换行符\n",
    "        eval_questions_en.append(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6209d314",
   "metadata": {},
   "source": [
    "最后，我们使用上面定义的函数运行评估，并通过 TruLens 获取并展示排行榜。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "d471cb1d-be4e-423b-a5a5-f216cbd8c9ff",
   "metadata": {
    "height": 81
   },
   "outputs": [],
   "source": [
    "import time\n",
    "# 定义运行评估的函数\n",
    "def run_evals(eval_questions, tru_recorder, query_engine):\n",
    "    for question in eval_questions:\n",
    "        with tru_recorder as recording:\n",
    "            response = query_engine.query(question)\n",
    "            time.sleep(2)\n",
    "            # 此处运行评估并记录结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "eea6f236-fe6a-470f-82b5-1a1e3d2593ce",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: d33cd5ff-82dc-4ba8-a6d7-c3b77f35315b.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 3/13\n",
      "本体论将知识表示为⼀个领域内的⼀\n",
      "组概...\n",
      "\n",
      "> Merging 4 nodes into parent node.\n",
      "> Parent node id: 4a656a65-c52b-410b-8152-1545952c47ad.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 6/13智能是否可以使⽤⾼级符号表达，如词和想法...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: cc7c42fb-22be-4ba5-b8c4-281f4b8b6236.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 4/13\n",
      "Kismet，⼀个具有表情等社交能⼒\n",
      "...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: fded6622-1120-4fae-8784-fe5052059204.\n",
      "> Parent node text: 2017年6⽉份马云在美国底特律举⾏“链接世界”（Gateway 17）产业⼤会，会上提出⼈⼯智能可能导致第三次世界⼤战，因为前两次产业⾰命都导致两次⼤\n",
      "战，战争原因并⾮这些创新发明本⾝，⽽是发...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: d6a24814-902b-4670-9bad-70f7a5be9f03.\n",
      "> Parent node text: The Behavioral and Brain Sciences, vol. 3, 1980）\n",
      "关于强⼈⼯智能的争论，不同于更⼴义的⼀元论和⼆元论的争论。其争论要点是：如果⼀台机器的唯⼀⼯作原...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: fded6622-1120-4fae-8784-fe5052059204.\n",
      "> Parent node text: 2017年6⽉份马云在美国底特律举⾏“链接世界”（Gateway 17）产业⼤会，会上提出⼈⼯智能可能导致第三次世界⼤战，因为前两次产业⾰命都导致两次⼤\n",
      "战，战争原因并⾮这些创新发明本⾝，⽽是发...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 8dd15e3b-2094-4fb8-9a72-c621ee342481.\n",
      "> Parent node text: 其它关于 动物 或其它⼈造系统的智能也普遍被认为是⼈⼯智能相关的研究课题。\n",
      "⼈⼯智能⽬前在 电脑 领域内，得到了愈加⼴泛的发挥。并在 机器⼈ 、经济政治决策、 控制系统 、仿真系统 中得到应⽤。...\n",
      "\n",
      "> Merging 4 nodes into parent node.\n",
      "> Parent node id: d33cd5ff-82dc-4ba8-a6d7-c3b77f35315b.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 3/13\n",
      "本体论将知识表示为⼀个领域内的⼀\n",
      "组概...\n",
      "\n",
      "> Merging 4 nodes into parent node.\n",
      "> Parent node id: e72ee5b0-427f-475e-8142-69703f6d2c05.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 5/133. 把AI当作同事，形成协同合作的团队...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: 12551bc8-6bdd-4504-b2e5-2b572d628504.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称\n",
      "中国⼤陆⼈⼯...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: d6a24814-902b-4670-9bad-70f7a5be9f03.\n",
      "> Parent node text: The Behavioral and Brain Sciences, vol. 3, 1980）\n",
      "关于强⼈⼯智能的争论，不同于更⼴义的⼀元论和⼆元论的争论。其争论要点是：如果⼀台机器的唯⼀⼯作原...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: d05268fc-e638-4cf4-aa06-bb827ff13a27.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 8/13东映特摄剧《 假⾯骑⼠Z ERO-ONE...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: cc7c42fb-22be-4ba5-b8c4-281f4b8b6236.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 4/13\n",
      "Kismet，⼀个具有表情等社交能⼒\n",
      "...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 8dd15e3b-2094-4fb8-9a72-c621ee342481.\n",
      "> Parent node text: 其它关于 动物 或其它⼈造系统的智能也普遍被认为是⼈⼯智能相关的研究课题。\n",
      "⼈⼯智能⽬前在 电脑 领域内，得到了愈加⼴泛的发挥。并在 机器⼈ 、经济政治决策、 控制系统 、仿真系统 中得到应⽤。...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: d6a24814-902b-4670-9bad-70f7a5be9f03.\n",
      "> Parent node text: The Behavioral and Brain Sciences, vol. 3, 1980）\n",
      "关于强⼈⼯智能的争论，不同于更⼴义的⼀元论和⼆元论的争论。其争论要点是：如果⼀台机器的唯⼀⼯作原...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: 12551bc8-6bdd-4504-b2e5-2b572d628504.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称\n",
      "中国⼤陆⼈⼯...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: 12551bc8-6bdd-4504-b2e5-2b572d628504.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称\n",
      "中国⼤陆⼈⼯...\n",
      "\n",
      "> Merging 4 nodes into parent node.\n",
      "> Parent node id: e72ee5b0-427f-475e-8142-69703f6d2c05.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 5/133. 把AI当作同事，形成协同合作的团队...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: d6a24814-902b-4670-9bad-70f7a5be9f03.\n",
      "> Parent node text: The Behavioral and Brain Sciences, vol. 3, 1980）\n",
      "关于强⼈⼯智能的争论，不同于更⼴义的⼀元论和⼆元论的争论。其争论要点是：如果⼀台机器的唯⼀⼯作原...\n",
      "\n",
      "> Merging 4 nodes into parent node.\n",
      "> Parent node id: 12551bc8-6bdd-4504-b2e5-2b572d628504.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称\n",
      "中国⼤陆⼈⼯...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: 4a656a65-c52b-410b-8152-1545952c47ad.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 6/13智能是否可以使⽤⾼级符号表达，如词和想法...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 8dd15e3b-2094-4fb8-9a72-c621ee342481.\n",
      "> Parent node text: 其它关于 动物 或其它⼈造系统的智能也普遍被认为是⼈⼯智能相关的研究课题。\n",
      "⼈⼯智能⽬前在 电脑 领域内，得到了愈加⼴泛的发挥。并在 机器⼈ 、经济政治决策、 控制系统 、仿真系统 中得到应⽤。...\n",
      "\n",
      "> Merging 4 nodes into parent node.\n",
      "> Parent node id: 12551bc8-6bdd-4504-b2e5-2b572d628504.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称\n",
      "中国⼤陆⼈⼯...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: fded6622-1120-4fae-8784-fe5052059204.\n",
      "> Parent node text: 2017年6⽉份马云在美国底特律举⾏“链接世界”（Gateway 17）产业⼤会，会上提出⼈⼯智能可能导致第三次世界⼤战，因为前两次产业⾰命都导致两次⼤\n",
      "战，战争原因并⾮这些创新发明本⾝，⽽是发...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 8dd15e3b-2094-4fb8-9a72-c621ee342481.\n",
      "> Parent node text: 其它关于 动物 或其它⼈造系统的智能也普遍被认为是⼈⼯智能相关的研究课题。\n",
      "⼈⼯智能⽬前在 电脑 领域内，得到了愈加⼴泛的发挥。并在 机器⼈ 、经济政治决策、 控制系统 、仿真系统 中得到应⽤。...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: fded6622-1120-4fae-8784-fe5052059204.\n",
      "> Parent node text: 2017年6⽉份马云在美国底特律举⾏“链接世界”（Gateway 17）产业⼤会，会上提出⼈⼯智能可能导致第三次世界⼤战，因为前两次产业⾰命都导致两次⼤\n",
      "战，战争原因并⾮这些创新发明本⾝，⽽是发...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: 12551bc8-6bdd-4504-b2e5-2b572d628504.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称\n",
      "中国⼤陆⼈⼯...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: d6a24814-902b-4670-9bad-70f7a5be9f03.\n",
      "> Parent node text: The Behavioral and Brain Sciences, vol. 3, 1980）\n",
      "关于强⼈⼯智能的争论，不同于更⼴义的⼀元论和⼆元论的争论。其争论要点是：如果⼀台机器的唯⼀⼯作原...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: fded6622-1120-4fae-8784-fe5052059204.\n",
      "> Parent node text: 2017年6⽉份马云在美国底特律举⾏“链接世界”（Gateway 17）产业⼤会，会上提出⼈⼯智能可能导致第三次世界⼤战，因为前两次产业⾰命都导致两次⼤\n",
      "战，战争原因并⾮这些创新发明本⾝，⽽是发...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: d6a24814-902b-4670-9bad-70f7a5be9f03.\n",
      "> Parent node text: The Behavioral and Brain Sciences, vol. 3, 1980）\n",
      "关于强⼈⼯智能的争论，不同于更⼴义的⼀元论和⼆元论的争论。其争论要点是：如果⼀台机器的唯⼀⼯作原...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: 4a656a65-c52b-410b-8152-1545952c47ad.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 6/13智能是否可以使⽤⾼级符号表达，如词和想法...\n",
      "\n",
      "> Merging 5 nodes into parent node.\n",
      "> Parent node id: 4a656a65-c52b-410b-8152-1545952c47ad.\n",
      "> Parent node text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 6/13智能是否可以使⽤⾼级符号表达，如词和想法...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 54c0b611-afdd-455f-ad8d-323bdac93a92.\n",
      "> Parent node text: 这和其他的⼦符号⽅法，如模糊控制和进化计算，都属于计算智能学科研\n",
      "究范畴[59]。\n",
      "1990年代，⼈⼯智能研究发展出复杂的数学⼯具来解决特定的分⽀问题。这些⼯具是真正的科学⽅法，即这些⽅法的结果...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 76fbc499-4fc9-41b4-b14e-090faa72952d.\n",
      "> Parent node text: ⽆论哪种学习⽅法都会进⾏误差分析，从⽽知道所提的⽅法在理论上是否误差有上限。\n",
      "⾃然语⾔处理探讨如何处理及运⽤⾃然语⾔，⾃然语⾔认知则是指让电脑“懂”⼈类的语⾔。⾃然语⾔⽣成系统把计算机数据转化为...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 对两层结构进行评估\n",
    "run_evals(eval_questions, tru_recorder, auto_merging_engine_0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "adf7ea31",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 95c8fa62-58cc-46a9-b08c-77fc05473471.\n",
      "> Parent node text: PAGE 26If you’re considering a role switch, a startup can be an easier place to do it than a big ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: db5ac140-f84a-4b8f-89c3-e4e47887874b.\n",
      "> Parent node text: PAGE 25Finding a job has a few predictable steps that include selecting the companies to which yo...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6abfebf1-807f-4961-925c-4506de715664.\n",
      "> Parent node text: PAGE 29If you’re preparing to switch roles (say, taking a job as a machine learning engineer for ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e0b6fee-493b-4a04-8a15-1d72c2c36e1d.\n",
      "> Parent node text: PAGE 19Develop a side hustle. Even if you have a full-time job, a fun project that may or may not...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 304c92a4-7169-4c00-a074-a293ac0cc32e.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 89264c70-af58-4055-83bc-c1bef94fe799.\n",
      "> Parent node text: PAGE 30Finding someone to interview isn’t always easy, but many people who are in senior position...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: f7fdeb9f-592b-4dd2-8860-4de8f4d1d35d.\n",
      "> Parent node text: PAGE 27There’s a lot we don’t know about the future: When will we cure Alzheimer’s disease? Who w...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: f6fbda9b-9353-4bc7-8869-6760f8aeae27.\n",
      "> Parent node text: PAGE 20Working on projects requires making tough choices about what to build and how to go \n",
      "about...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e0b6fee-493b-4a04-8a15-1d72c2c36e1d.\n",
      "> Parent node text: PAGE 19Develop a side hustle. Even if you have a full-time job, a fun project that may or may not...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 880bd756-51fd-4297-8a0d-212f72d3ac57.\n",
      "> Parent node text: PAGE 24A Simple Framework \n",
      "for Starting Your AI \n",
      "Job SearchCHAPTER 7\n",
      "JOBS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6abfebf1-807f-4961-925c-4506de715664.\n",
      "> Parent node text: PAGE 29If you’re preparing to switch roles (say, taking a job as a machine learning engineer for ...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: f6fbda9b-9353-4bc7-8869-6760f8aeae27.\n",
      "> Parent node text: PAGE 20Working on projects requires making tough choices about what to build and how to go \n",
      "about...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6abfebf1-807f-4961-925c-4506de715664.\n",
      "> Parent node text: PAGE 29If you’re preparing to switch roles (say, taking a job as a machine learning engineer for ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: f7fdeb9f-592b-4dd2-8860-4de8f4d1d35d.\n",
      "> Parent node text: PAGE 27There’s a lot we don’t know about the future: When will we cure Alzheimer’s disease? Who w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e0b6fee-493b-4a04-8a15-1d72c2c36e1d.\n",
      "> Parent node text: PAGE 19Develop a side hustle. Even if you have a full-time job, a fun project that may or may not...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 82324caa-53eb-43db-88e3-41999292befd.\n",
      "> Parent node text: PAGE 17Finding Projects that \n",
      "Complement Your \n",
      "Career GoalsCHAPTER 5\n",
      "PROJECTS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 304c92a4-7169-4c00-a074-a293ac0cc32e.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6548dd32-96ca-4041-928f-88a052e642b4.\n",
      "> Parent node text: PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? I...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: d349af76-66b2-47a5-841a-bb333ee3a17e.\n",
      "> Parent node text: PAGE 9In the previous chapter, I introduced three key steps for building a career in AI: learning...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 108d8e75-c878-4cbe-91cd-7a285057700b.\n",
      "> Parent node text: PAGE 4Coding AI Is the New Literacy\n",
      "Today we take it for granted that many people know how to rea...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 81cd1121-06fe-4bbf-b47a-20ff14269ef4.\n",
      "> Parent node text: PAGE 10This is a lot to learn!\n",
      "Even after you master everything on this list, I hope you’ll keep ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: d36513e5-d7f1-4700-9db1-8d83845aec3e.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 8959ebcc-0e7d-4e3d-9770-2664b8400cb8.\n",
      "> Parent node text: PAGE 12Should You \n",
      "Learn Math to \n",
      "Get a Job in AI? CHAPTER 3\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 81cd1121-06fe-4bbf-b47a-20ff14269ef4.\n",
      "> Parent node text: PAGE 10This is a lot to learn!\n",
      "Even after you master everything on this list, I hope you’ll keep ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c1583f84-c6d4-4c8d-a08b-49e0a4954794.\n",
      "> Parent node text: PAGE 11\n",
      "The Best Way to Build \n",
      "a New Habit\n",
      "One of my favorite books is BJ Fogg’s, Tiny Habits: Th...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 924402c7-d26d-402c-9d53-2e327e5ac40e.\n",
      "> Parent node text: PAGE 8Learning Technical \n",
      "Skills for a Promising \n",
      "AI CareerCHAPTER 2\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6abfebf1-807f-4961-925c-4506de715664.\n",
      "> Parent node text: PAGE 29If you’re preparing to switch roles (say, taking a job as a machine learning engineer for ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 95c8fa62-58cc-46a9-b08c-77fc05473471.\n",
      "> Parent node text: PAGE 26If you’re considering a role switch, a startup can be an easier place to do it than a big ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: db5ac140-f84a-4b8f-89c3-e4e47887874b.\n",
      "> Parent node text: PAGE 25Finding a job has a few predictable steps that include selecting the companies to which yo...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 89264c70-af58-4055-83bc-c1bef94fe799.\n",
      "> Parent node text: PAGE 30Finding someone to interview isn’t always easy, but many people who are in senior position...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 89264c70-af58-4055-83bc-c1bef94fe799.\n",
      "> Parent node text: PAGE 30Finding someone to interview isn’t always easy, but many people who are in senior position...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e0b6fee-493b-4a04-8a15-1d72c2c36e1d.\n",
      "> Parent node text: PAGE 19Develop a side hustle. Even if you have a full-time job, a fun project that may or may not...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6abfebf1-807f-4961-925c-4506de715664.\n",
      "> Parent node text: PAGE 29If you’re preparing to switch roles (say, taking a job as a machine learning engineer for ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 89264c70-af58-4055-83bc-c1bef94fe799.\n",
      "> Parent node text: PAGE 30Finding someone to interview isn’t always easy, but many people who are in senior position...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 21c8d7be-7c3c-48dd-9b96-27ccc0e706c6.\n",
      "> Parent node text: PAGE 28Using Informational \n",
      "Interviews to Find \n",
      "the Right JobCHAPTER 8\n",
      "JOBS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 880bd756-51fd-4297-8a0d-212f72d3ac57.\n",
      "> Parent node text: PAGE 24A Simple Framework \n",
      "for Starting Your AI \n",
      "Job SearchCHAPTER 7\n",
      "JOBS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: db5ac140-f84a-4b8f-89c3-e4e47887874b.\n",
      "> Parent node text: PAGE 25Finding a job has a few predictable steps that include selecting the companies to which yo...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 83651026-2f44-423e-a72c-c93117348111.\n",
      "> Parent node text: PAGE 32In this chapter, I’d like to \n",
      "discuss some fine points \n",
      "of finding a job.\n",
      "The typical job ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 21c8d7be-7c3c-48dd-9b96-27ccc0e706c6.\n",
      "> Parent node text: PAGE 28Using Informational \n",
      "Interviews to Find \n",
      "the Right JobCHAPTER 8\n",
      "JOBS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 880bd756-51fd-4297-8a0d-212f72d3ac57.\n",
      "> Parent node text: PAGE 24A Simple Framework \n",
      "for Starting Your AI \n",
      "Job SearchCHAPTER 7\n",
      "JOBS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: daaa89b7-3df6-43df-a0c5-7e4d4864636a.\n",
      "> Parent node text: PAGE 31Finding the Right \n",
      "AI Job for YouCHAPTER 9\n",
      "JOBS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: ddb56829-91b0-4941-8147-cd04e7311219.\n",
      "> Parent node text: PAGE 34Keys to Building a \n",
      "Career in AICHAPTER 10\n",
      "JOBS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 89264c70-af58-4055-83bc-c1bef94fe799.\n",
      "> Parent node text: PAGE 30Finding someone to interview isn’t always easy, but many people who are in senior position...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6abfebf1-807f-4961-925c-4506de715664.\n",
      "> Parent node text: PAGE 29If you’re preparing to switch roles (say, taking a job as a machine learning engineer for ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 924402c7-d26d-402c-9d53-2e327e5ac40e.\n",
      "> Parent node text: PAGE 8Learning Technical \n",
      "Skills for a Promising \n",
      "AI CareerCHAPTER 2\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: f7fdeb9f-592b-4dd2-8860-4de8f4d1d35d.\n",
      "> Parent node text: PAGE 27There’s a lot we don’t know about the future: When will we cure Alzheimer’s disease? Who w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 304c92a4-7169-4c00-a074-a293ac0cc32e.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c9ba0db2-22e9-4260-a0d2-a594a347876a.\n",
      "> Parent node text: PAGE 21Building a Portfolio of \n",
      "Projects that Shows \n",
      "Skill Progression CHAPTER 6\n",
      "PROJECTS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6548dd32-96ca-4041-928f-88a052e642b4.\n",
      "> Parent node text: PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? I...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: db5ac140-f84a-4b8f-89c3-e4e47887874b.\n",
      "> Parent node text: PAGE 25Finding a job has a few predictable steps that include selecting the companies to which yo...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 95c8fa62-58cc-46a9-b08c-77fc05473471.\n",
      "> Parent node text: PAGE 26If you’re considering a role switch, a startup can be an easier place to do it than a big ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6abfebf1-807f-4961-925c-4506de715664.\n",
      "> Parent node text: PAGE 29If you’re preparing to switch roles (say, taking a job as a machine learning engineer for ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 83651026-2f44-423e-a72c-c93117348111.\n",
      "> Parent node text: PAGE 32In this chapter, I’d like to \n",
      "discuss some fine points \n",
      "of finding a job.\n",
      "The typical job ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: f7fdeb9f-592b-4dd2-8860-4de8f4d1d35d.\n",
      "> Parent node text: PAGE 27There’s a lot we don’t know about the future: When will we cure Alzheimer’s disease? Who w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 89264c70-af58-4055-83bc-c1bef94fe799.\n",
      "> Parent node text: PAGE 30Finding someone to interview isn’t always easy, but many people who are in senior position...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: d36513e5-d7f1-4700-9db1-8d83845aec3e.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 212868af-70b1-4eab-ace2-829b078974e5.\n",
      "> Parent node text: PAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 94031f2f-c7c2-4c3b-b523-1f65493d963f.\n",
      "> Parent node text: PAGE 37Overcoming Imposter \n",
      "SyndromeCHAPTER 11\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 304c92a4-7169-4c00-a074-a293ac0cc32e.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 89264c70-af58-4055-83bc-c1bef94fe799.\n",
      "> Parent node text: PAGE 30Finding someone to interview isn’t always easy, but many people who are in senior position...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 924402c7-d26d-402c-9d53-2e327e5ac40e.\n",
      "> Parent node text: PAGE 8Learning Technical \n",
      "Skills for a Promising \n",
      "AI CareerCHAPTER 2\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9771ef5e-b1b1-4f2b-9e6d-ceafb415f5aa.\n",
      "> Parent node text: PAGE 40Make Every Day CountFinal Thoughts\n",
      "Every year on my birthday, I get to thinking about the ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c1583f84-c6d4-4c8d-a08b-49e0a4954794.\n",
      "> Parent node text: PAGE 11\n",
      "The Best Way to Build \n",
      "a New Habit\n",
      "One of my favorite books is BJ Fogg’s, Tiny Habits: Th...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: f7fdeb9f-592b-4dd2-8860-4de8f4d1d35d.\n",
      "> Parent node text: PAGE 27There’s a lot we don’t know about the future: When will we cure Alzheimer’s disease? Who w...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: f6fbda9b-9353-4bc7-8869-6760f8aeae27.\n",
      "> Parent node text: PAGE 20Working on projects requires making tough choices about what to build and how to go \n",
      "about...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 108d8e75-c878-4cbe-91cd-7a285057700b.\n",
      "> Parent node text: PAGE 4Coding AI Is the New Literacy\n",
      "Today we take it for granted that many people know how to rea...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 988133b8-bd97-40e4-a6db-6c4073d3604a.\n",
      "> Parent node text: PAGE 2\"AI is the new \n",
      "electricity. It will \n",
      "transform and improve \n",
      "all areas of human life.\"\n",
      "Andr...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6548dd32-96ca-4041-928f-88a052e642b4.\n",
      "> Parent node text: PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? I...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 21c8d7be-7c3c-48dd-9b96-27ccc0e706c6.\n",
      "> Parent node text: PAGE 28Using Informational \n",
      "Interviews to Find \n",
      "the Right JobCHAPTER 8\n",
      "JOBS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6548dd32-96ca-4041-928f-88a052e642b4.\n",
      "> Parent node text: PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? I...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: d349af76-66b2-47a5-841a-bb333ee3a17e.\n",
      "> Parent node text: PAGE 9In the previous chapter, I introduced three key steps for building a career in AI: learning...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: d36513e5-d7f1-4700-9db1-8d83845aec3e.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 108d8e75-c878-4cbe-91cd-7a285057700b.\n",
      "> Parent node text: PAGE 4Coding AI Is the New Literacy\n",
      "Today we take it for granted that many people know how to rea...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 81cd1121-06fe-4bbf-b47a-20ff14269ef4.\n",
      "> Parent node text: PAGE 10This is a lot to learn!\n",
      "Even after you master everything on this list, I hope you’ll keep ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9ece81e9-ccc2-423f-9101-88c993bc6102.\n",
      "> Parent node text: PAGE 14Scoping Successful \n",
      "AI ProjectsCHAPTER 4\n",
      "PROJECTS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 82324caa-53eb-43db-88e3-41999292befd.\n",
      "> Parent node text: PAGE 17Finding Projects that \n",
      "Complement Your \n",
      "Career GoalsCHAPTER 5\n",
      "PROJECTS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: a0159f01-178e-442f-b6e1-aa8cdaf2f6aa.\n",
      "> Parent node text: PAGE 5Three Steps to \n",
      "Career GrowthCHAPTER 1\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e0b6fee-493b-4a04-8a15-1d72c2c36e1d.\n",
      "> Parent node text: PAGE 19Develop a side hustle. Even if you have a full-time job, a fun project that may or may not...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 924402c7-d26d-402c-9d53-2e327e5ac40e.\n",
      "> Parent node text: PAGE 8Learning Technical \n",
      "Skills for a Promising \n",
      "AI CareerCHAPTER 2\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e0b6fee-493b-4a04-8a15-1d72c2c36e1d.\n",
      "> Parent node text: PAGE 19Develop a side hustle. Even if you have a full-time job, a fun project that may or may not...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 82324caa-53eb-43db-88e3-41999292befd.\n",
      "> Parent node text: PAGE 17Finding Projects that \n",
      "Complement Your \n",
      "Career GoalsCHAPTER 5\n",
      "PROJECTS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 304c92a4-7169-4c00-a074-a293ac0cc32e.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6548dd32-96ca-4041-928f-88a052e642b4.\n",
      "> Parent node text: PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? I...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e0b6fee-493b-4a04-8a15-1d72c2c36e1d.\n",
      "> Parent node text: PAGE 19Develop a side hustle. Even if you have a full-time job, a fun project that may or may not...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 82324caa-53eb-43db-88e3-41999292befd.\n",
      "> Parent node text: PAGE 17Finding Projects that \n",
      "Complement Your \n",
      "Career GoalsCHAPTER 5\n",
      "PROJECTS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9ece81e9-ccc2-423f-9101-88c993bc6102.\n",
      "> Parent node text: PAGE 14Scoping Successful \n",
      "AI ProjectsCHAPTER 4\n",
      "PROJECTS\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 108d8e75-c878-4cbe-91cd-7a285057700b.\n",
      "> Parent node text: PAGE 4Coding AI Is the New Literacy\n",
      "Today we take it for granted that many people know how to rea...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6548dd32-96ca-4041-928f-88a052e642b4.\n",
      "> Parent node text: PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? I...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 8959ebcc-0e7d-4e3d-9770-2664b8400cb8.\n",
      "> Parent node text: PAGE 12Should You \n",
      "Learn Math to \n",
      "Get a Job in AI? CHAPTER 3\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: d36513e5-d7f1-4700-9db1-8d83845aec3e.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 988133b8-bd97-40e4-a6db-6c4073d3604a.\n",
      "> Parent node text: PAGE 2\"AI is the new \n",
      "electricity. It will \n",
      "transform and improve \n",
      "all areas of human life.\"\n",
      "Andr...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 212868af-70b1-4eab-ace2-829b078974e5.\n",
      "> Parent node text: PAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 95c8fa62-58cc-46a9-b08c-77fc05473471.\n",
      "> Parent node text: PAGE 26If you’re considering a role switch, a startup can be an easier place to do it than a big ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 8959ebcc-0e7d-4e3d-9770-2664b8400cb8.\n",
      "> Parent node text: PAGE 12Should You \n",
      "Learn Math to \n",
      "Get a Job in AI? CHAPTER 3\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: a0159f01-178e-442f-b6e1-aa8cdaf2f6aa.\n",
      "> Parent node text: PAGE 5Three Steps to \n",
      "Career GrowthCHAPTER 1\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 304c92a4-7169-4c00-a074-a293ac0cc32e.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 212868af-70b1-4eab-ace2-829b078974e5.\n",
      "> Parent node text: PAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 212868af-70b1-4eab-ace2-829b078974e5.\n",
      "> Parent node text: PAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 304c92a4-7169-4c00-a074-a293ac0cc32e.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 8959ebcc-0e7d-4e3d-9770-2664b8400cb8.\n",
      "> Parent node text: PAGE 12Should You \n",
      "Learn Math to \n",
      "Get a Job in AI? CHAPTER 3\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: d349af76-66b2-47a5-841a-bb333ee3a17e.\n",
      "> Parent node text: PAGE 9In the previous chapter, I introduced three key steps for building a career in AI: learning...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6548dd32-96ca-4041-928f-88a052e642b4.\n",
      "> Parent node text: PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? I...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 81cd1121-06fe-4bbf-b47a-20ff14269ef4.\n",
      "> Parent node text: PAGE 10This is a lot to learn!\n",
      "Even after you master everything on this list, I hope you’ll keep ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 108d8e75-c878-4cbe-91cd-7a285057700b.\n",
      "> Parent node text: PAGE 4Coding AI Is the New Literacy\n",
      "Today we take it for granted that many people know how to rea...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0739e57d-3834-4794-b479-25971c8cceea.\n",
      "> Parent node text: PAGE 22Over the course of a career, you’re likely to work on projects in succession, each growing...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6abfebf1-807f-4961-925c-4506de715664.\n",
      "> Parent node text: PAGE 29If you’re preparing to switch roles (say, taking a job as a machine learning engineer for ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 5f7f8a9b-96da-41be-b906-e9e52084b647.\n",
      "> Parent node text: PAGE 15One of the most important skills of an AI architect is the ability to identify ideas that ...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 304c92a4-7169-4c00-a074-a293ac0cc32e.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 52d5d005-7105-408e-851e-213942a04dd0.\n",
      "> Parent node text: PAGE 16Determine milestones. Once you’ve deemed a project sufficiently \n",
      "valuable, the next step i...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 34343c7a-bbc6-4d62-8dca-cea4fab0cdf0.\n",
      "> Parent node text: PAGE 23Each project is only one step on a longer journey, hopefully one that has a positive impac...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6548dd32-96ca-4041-928f-88a052e642b4.\n",
      "> Parent node text: PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? I...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 0e2d88a0-d0af-4efd-97dd-19a0a45d7a44.\n",
      "> Parent node text: PAGE 6The rapid rise of AI has led to a rapid rise in AI jobs, and many people are building excit...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 8959ebcc-0e7d-4e3d-9770-2664b8400cb8.\n",
      "> Parent node text: PAGE 12Should You \n",
      "Learn Math to \n",
      "Get a Job in AI? CHAPTER 3\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 924402c7-d26d-402c-9d53-2e327e5ac40e.\n",
      "> Parent node text: PAGE 8Learning Technical \n",
      "Skills for a Promising \n",
      "AI CareerCHAPTER 2\n",
      "LEARNING\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: d36513e5-d7f1-4700-9db1-8d83845aec3e.\n",
      "> Parent node text: PAGE 38Before we dive into the final chapter of this book, I’d like to address the serious matter...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 94031f2f-c7c2-4c3b-b523-1f65493d963f.\n",
      "> Parent node text: PAGE 37Overcoming Imposter \n",
      "SyndromeCHAPTER 11\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: c52f5841-04c2-4e9b-8aa3-4f081ae1d223.\n",
      "> Parent node text: PAGE 3Table of \n",
      "ContentsIntroduction: Coding AI is the New Literacy.\n",
      "Chapter 1: Three Steps to Ca...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 212868af-70b1-4eab-ace2-829b078974e5.\n",
      "> Parent node text: PAGE 39My three-year-old daughter (who can barely count to 12) regularly tries to teach things to...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 304c92a4-7169-4c00-a074-a293ac0cc32e.\n",
      "> Parent node text: PAGE 35Keys to Building a Career in AI CHAPTER 10\n",
      "The path to career success in AI is more comple...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 20225074-fd66-4387-bc5e-99921ba84edb.\n",
      "> Parent node text: PAGE 7These phases apply in a wide \n",
      "range of professions, but AI \n",
      "involves unique elements.\n",
      "For e...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 108d8e75-c878-4cbe-91cd-7a285057700b.\n",
      "> Parent node text: PAGE 4Coding AI Is the New Literacy\n",
      "Today we take it for granted that many people know how to rea...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 71104f5e-d17b-4017-9cdc-a4439e918124.\n",
      "> Parent node text: PAGE 36Keys to Building a Career in AI CHAPTER 10\n",
      "Of all the steps in building a career, this \n",
      "on...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 9145b623-e40f-4038-95af-d99ee4d6cba6.\n",
      "> Parent node text: PAGE 33Choose who to work with. It’s tempting to take a position because of the projects you’ll w...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 6548dd32-96ca-4041-928f-88a052e642b4.\n",
      "> Parent node text: PAGE 13Should you Learn Math to Get a Job in AI? CHAPTER 3\n",
      "Is math a foundational skill for AI? I...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 75a5c366-ccda-4712-8525-d8dc14abb04b.\n",
      "> Parent node text: PAGE 18It goes without saying that we should only work on projects that are responsible, ethical,...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 988133b8-bd97-40e4-a6db-6c4073d3604a.\n",
      "> Parent node text: PAGE 2\"AI is the new \n",
      "electricity. It will \n",
      "transform and improve \n",
      "all areas of human life.\"\n",
      "Andr...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 对两层结构进行评估\n",
    "run_evals(eval_questions_en, tru_recorder_en, auto_merging_engine_0_en)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6c30160",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "54cc6991-e743-4e33-9e22-414362a6aae0",
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Answer Relevance</th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>app_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>app_0_en</th>\n",
       "      <td>0.979167</td>\n",
       "      <td>0.746528</td>\n",
       "      <td>0.413333</td>\n",
       "      <td>6.5</td>\n",
       "      <td>0.00375</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Answer Relevance  Groundedness  Context Relevance  latency  \\\n",
       "app_id                                                                 \n",
       "app_0_en          0.979167      0.746528           0.413333      6.5   \n",
       "\n",
       "          total_cost  \n",
       "app_id                \n",
       "app_0_en     0.00375  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from trulens_eval import Tru\n",
    "\n",
    "# 获取并显示排行榜\n",
    "Tru().get_leaderboard(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f204af1a-ab10-4e50-b2d6-ba78e2a66e90",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting dashboard ...\n",
      "Config file already exists. Skipping writing process.\n",
      "Credentials file already exists. Skipping writing process.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d05ed08c40847299da2215bb70f7058",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Accordion(children=(VBox(children=(VBox(children=(Label(value='STDOUT'), Output())), VBox(children=(Label(valu…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dashboard started at http://192.168.2.7:8501 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Popen: returncode: None args: ['streamlit', 'run', '--server.headless=True'...>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 运行TruLens仪表板查看详细评估结果\n",
    "Tru().run_dashboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c36721ff-9fe4-43e9-b7ed-435f6d054e29",
   "metadata": {},
   "source": [
    "### 3.2、三层结构的索引构建"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33dc1674",
   "metadata": {},
   "source": [
    "接下来，我们将尝试一个更复杂的自动合并索引构建方法，包含三层结构，并可能提高检索的质量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "5a575af9-37b8-442a-b6f2-3ed04bf742f1",
   "metadata": {
    "height": 132
   },
   "outputs": [],
   "source": [
    "# 构建具有三层结构的自动合并索引\n",
    "auto_merging_index_1 = build_automerging_index(\n",
    "    documents,\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",\n",
    "    save_dir=\"merging_index_1\",\n",
    "    chunk_sizes=[2048, 512, 128],  # 三层结构的块大小设置\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "99f18b15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 构建具有三层结构的自动合并索引\n",
    "auto_merging_index_1_en = build_automerging_index(\n",
    "    documents_en,\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",\n",
    "    save_dir=\"merging_index_1_en\",\n",
    "    chunk_sizes=[2048, 512, 128],  # 三层结构的块大小设置\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2487f2a",
   "metadata": {},
   "source": [
    "为这个三层结构的自动合并索引设置查询引擎。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "b0493fb6-3a5b-4d14-81ba-f26d9e4d46b3",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "# 获取三层结构的自动合并查询引擎\n",
    "auto_merging_engine_1 = get_automerging_query_engine(\n",
    "    auto_merging_index_1,\n",
    "    similarity_top_k=12,\n",
    "    rerank_top_n=6,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "18e66fc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取三层结构的自动合并查询引擎\n",
    "auto_merging_engine_1_en = get_automerging_query_engine(\n",
    "    auto_merging_index_1_en,\n",
    "    similarity_top_k=12,\n",
    "    rerank_top_n=6,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9118a28e",
   "metadata": {},
   "source": [
    "同样，我们定义一个 TruLens 记录器，并用相同的评估问题运行评估。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "d3198fa2-6a81-4edd-a0c8-843c3838f56f",
   "metadata": {
    "height": 81
   },
   "outputs": [],
   "source": [
    "# 为三层结构准备TruLens记录器\n",
    "tru_recorder = get_prebuilt_trulens_recorder(\n",
    "    auto_merging_engine_1,\n",
    "    app_id ='app_1'  # 设置应用ID\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52163adf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 为三层结构准备TruLens记录器\n",
    "tru_recorder_en = get_prebuilt_trulens_recorder(\n",
    "    auto_merging_engine_1_en,\n",
    "    app_id ='app_1_en'  # 设置应用ID\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "5063c989-9f96-4bfa-bb50-fa9dbf6d576c",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Merging 4 nodes into parent node.\n",
      "> Parent node id: 90b3476a-489a-4fee-beb7-1c5363a50ab7.\n",
      "> Parent node text: [16]\n",
      "⼈类解决问题的模式通常是⽤最快捷、直观的判断，⽽不是有意识的、⼀步⼀步的推导，早期⼈⼯智能研究通常使⽤逐步推导的⽅式。[17]⼈⼯智能研究已经\n",
      "于这种“次表征性的”解决问题⽅法获取进展...\n",
      "\n",
      "> Merging 2 nodes into parent node.\n",
      "> Parent node id: 7bab41fa-272e-4847-a568-e22dbc7e9fed.\n",
      "> Parent node text: 依⽬前的研究⽅向，电脑⽆法突变、苏醒、产⽣⾃我意志，AI也不可能具有创意与智能、同情⼼与审美等这⽅⾯的能⼒。\n",
      "AI逐渐普及后，将会在企业管理中扮演很重要的⾓⾊，⽽⼈类的管理者应 如何适度的调整⾃...\n",
      "\n",
      "> Merging 5 nodes into parent node.\n",
      "> Parent node id: b6df9f33-521b-434b-b26d-8f6876a93184.\n",
      "> Parent node text: 其它关于 动物 或其它⼈造系统的智能也普遍被认为是⼈⼯智能相关的研究课题。\n",
      "⼈⼯智能⽬前在 电脑 领域内，得到了愈加⼴泛的发挥。并在 机器⼈ 、经济政治决策、 控制系统 、仿真系统 中得到应⽤。...\n",
      "\n",
      "> Merging 3 nodes into parent node.\n",
      "> Parent node id: a7a0bda7-af15-46f6-a27f-28cf2b2357e0.\n",
      "> Parent node text: 2017年6⽉份马云在美国底特律举⾏“链接世界”（Gateway 17）产业⼤会，会上提出⼈⼯智能可能导致第三次世界⼤战，因为前两次产业⾰命都导致两次⼤\n",
      "战，战争原因并⾮这些创新发明本⾝，⽽是发...\n",
      "\n",
      "> Merging 1 nodes into parent node.\n",
      "> Parent node id: 93ed48c1-a43a-4ebe-ab72-46dd57662601.\n",
      "> Parent node text: 这与认知科学领域中的表征\n",
      "感知论点是⼀致的:更⾼的智能需要个体的表征（如移动，感知和形象）。\n",
      "计算智能：1 980年代中 David Rumelhart等再次提出 神经⽹络 和联结主义[58]。\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 对三层结构进行评估\n",
    "run_evals(eval_questions, tru_recorder, auto_merging_engine_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9258fdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 对三层结构进行评估\n",
    "run_evals(eval_questions_en, tru_recorder_en, auto_merging_engine_1_en)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e28fde7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "c26b5c94-16ce-415d-93ca-a131b05f903f",
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Answer Relevance</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>app_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>app_0</th>\n",
       "      <td>0.358788</td>\n",
       "      <td>0.888571</td>\n",
       "      <td>0.9400</td>\n",
       "      <td>7.866667</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>app_1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.9125</td>\n",
       "      <td>7.866667</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Context Relevance  Groundedness  Answer Relevance   latency  \\\n",
       "app_id                                                                \n",
       "app_0            0.358788      0.888571            0.9400  7.866667   \n",
       "app_1                 NaN           NaN            0.9125  7.866667   \n",
       "\n",
       "        total_cost  \n",
       "app_id              \n",
       "app_0          0.0  \n",
       "app_1          0.0  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from trulens_eval import Tru\n",
    "\n",
    "# 获取并显示排行榜\n",
    "Tru().get_leaderboard(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "3988b3f8-bd4c-41e5-be50-e90b8366f4bd",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting dashboard ...\n",
      "Config file already exists. Skipping writing process.\n",
      "Credentials file already exists. Skipping writing process.\n",
      "Dashboard already running at path:   Network URL: http://192.168.2.7:8501\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Popen: returncode: None args: ['streamlit', 'run', '--server.headless=True'...>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 运行TruLens仪表板查看详细评估结果\n",
    "Tru().run_dashboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d159edd",
   "metadata": {},
   "source": [
    "通过比较两层和三层结构的自动合并索引的 TruLens 评估结果，我们可以深入了解不同层次结构对检索性能的影响，并据此优化我们的自动合并策略。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
